System and method for tagging a region within a distributed video file

ABSTRACT

A system and method for tagging a region within a frame of a distributed video file to permit easier retrievability of the region of interest.

FIELD OF THE INVENTION

The present invention relates to a method and apparatus for annotatingand indexing electronic files with metadata, and navigating saidmetadata for retrieval and use by the user. More particularly theelectronic files described in this embodiment of the invention aremultimedia files, such as digital audio and video files, and themetadata applied to the files consists of both keywords and hierarchiesof classification terms. Further, the present invention provides amethod and apparatus for navigating large volumes of structuredinformation including hierarchies of classification terms. The presentinvention also provides for a collaborative online environment wherecommunities and groups can gather and exchange ideas through mash upsand the collaborative use of personal dictionaries.

BACKGROUND OF THE INVENTION

The desire for personal control, possession and ownership drove theevolution of computing from the handful of mainframe computers in largecorporate, government and university settings decades ago to thesituation today where personal computers now sit in homes and businesseseverywhere.

But the widespread availability and ownership of personal computers indeveloped countries did nothing to quell consumers' thirst for personal‘computing’ devices per se. Laptops, mobile phones, camera phones, gamesconsoles, portable games consoles, portable music players and PDA's haveproliferated in society alongside personal computers.

And in more recent times, as these kinds of devices have continued toproliferate, we have also seen major developments with software andservices on the web. In particular, we have seen the evolution frompeer-to-peer file sharing networks to social networks like LinkedIn andMySpace, to the explosion in video sharing websites like YouTube and MSNVideo. These have all played their part in the rise of what's becomeknown as social networking.

One of the major uses of the internet today is to disseminate, discuss,and otherwise consume multimedia content including videos, photos, andaudio tracks. This is being done today, not only on personal computers,but also handheld mobile computing devices such as PDA's, digital mobiletelephones, navigation devices, personal media players such as Apple'siPod, and indeed, devices which incorporate all of the features of thesehandheld devices such as the iPhone. Further, online multimedia contentis being delivered directly to set top boxes and games consolesconnected to the internet such as Tivo devices and Microsoft Xbox's.

One problem associated with accessing multimedia material on theinternet is that more often than not, the segment of interest is oftenfound within a larger multimedia file. For example, if a particularscene is found one hour into a movie, a link to that movie wouldgenerally take a user to the start of the movie and not to the sectionof interest.

Another problem associated with accessing multimedia material on theinternet is that metadata associated with the multimedia file, whichcould be useful to determine relevancy, is often ambiguous and of littlereal help in determining what the multimedia file contains.

Another problem with prior art annotation and retrieval systems is thatthey provide little scope for using natural language and/orclassification terms unique to the user. Ultimately there is littlescope for personalisation of the process.

Further problems associated with prior art systems is that they arerigid with respect to users. Some systems provide for single usersystems, whilst others are designed with groups in mind and are lessamenable for use by a single user or different types of users.

Further problems associated with prior art systems is that they lack theability to treat segments of multimedia files as entries in their ownright in a metadata database without first physically creating segmentsfor re-entry into the database. The creation of further copies ofportions of multimedia files is therefore subject to copyright laws andthe problems that are associated with them.

Further, some prior art systems lack the ability to associate databaseentries with logically segmented multimedia files.

Further problems with automated tools for extracting metadata fromcontent (such as voice to text, voice recognition and patternrecognition tools) are that they are limited in that they cannot capturethe range and depth of human interpretation. For example, does thefootage of an allied soldier in the desert depict a soldier, a Sergeant,a son, an infidel, a hero, an American, a voter or a military uniform?

Further, multimedia annotation based on keywords is devoid of context.This ability to apply and preserve context will become more and moreimportant as the volume of multimedia on the web increases. Context isimportant because keyword searches and searches involving combinationsof keywords return massive numbers of hits, making it extremelydifficult for users to identify the relevant file/s, and even moredifficult to identify the relevant segment or point within a file.Further, context is important to online advertisers who need to targetthe particular type content in which they want their advertisements toappear in connection with.

OBJECTION OF THE INVENTION

It is an object of the invention to overcome at least some of the stateddeficiencies of the prior art.

SUMMARY OF THE INVENTION

According to one aspect of the invention there is provided, a system forlocating, segmenting, annotating and retrieving multimedia files, saidsystem providing:

database of metadata relating to multimedia files;

database manager;

a database client for accessing the data contained within the database,said client providing:

-   -   deep linking functionality where the user can load, open and        start the playing of a multimedia file from a point intermediate        to its start and end point, and without the user having to        experience the portion between the start point and the        intermediate point;    -   segmentation functionality whereby large multimedia files can be        broken down into shorter segments or individual frames (or still        images) or regions within frames (or still images);    -   metadata annotation functionality wherein the user can annotate        whole multimedia files, individual segments, frames and regions        within frames with alphanumeric information;    -   retrieval functionality for browsing and searching the metadata;        and    -   access functionality for displaying and viewing the multimedia        file;    -   wherein the user annotates the multimedia file or segment of        multimedia file with metadata which is saved in the database,        and wherein the user desired to locate a multimedia file, said        metadata is searched or browsed to locate the database entry        associated with multimedia file in question.

In a preferred embodiment of the invention, the annotation of themultimedia file or portion thereof is achieved by recording in themetadata saved in the database, a hierarchy of classification terms thatform an information branch of the database's knowledge tree.

In a preferred embodiment of the invention the multimedia files arestored locally or are available to the database manager over an IPconnection.

In a further preferred embodiment of the invention the IP connection isa HTTP connection over TCP/IP.

In a preferred embodiment of the invention the database manager providesfor 2 or more types of user accounts.

In a further preferred embodiment of the invention the database managerprovides for administrator, librarian, annotator and searcher useraccounts.

In a further preferred embodiment of the invention there may bemultiples of each user type.

In a further embodiment of the invention the user is able to annotatethe multimedia file or segmented portion thereof within still video byselecting the lowermost classification term of the desired informationbranch.

In a further embodiment of the invention the user is able to select thelowermost classification term of the information branch by pointing,clicking and dragging the lowermost classification term to the relevantstoryboard or catalog entry.

In a further embodiment of the invention the user is able to createoverlapping segments of the one multimedia file by logically segmentingthe original file in a way such that the end point of the first segmentproceeds beyond the start point of the second segment.

In a preferred embodiment of the invention the database client is loadedonto a computing apparatus, said apparatus comprising at least one dataprocessor, input and output modules, memory coupled to the at least onedata processor and having stored therein programming instructions forperforming data processing.

In a preferred embodiment of the invention, the computing apparatusfurther comprises persistent storage.

In a preferred embodiment of the invention, the computing apparatusfurther comprises a network adaptor which permits the integration of thecomputing apparatus on an IP network.

In a preferred embodiment of the invention, the IP network is theInternet, provided over a TCP/IP connection.

In a preferred embodiment of the invention the computing device is apersonal computer or laptop.

In an another embodiment of the invention, the computing apparatus canbe taken from the following list of devices: network workstations, settop boxes, digital mobile telephones, network IP routers, media players,games machines, NAS devices, video conferencing equipment and satellitenavigation displays.

According to a further aspect of the invention there is provided amethod for locating, segmenting and annotating multimedia files, saidmethod comprising the following steps:

identifying the name and location of a multimedia file;

creating a database entry for associated metadata;

loading the multimedia file into memory;

assigning a short title to the database entry;

assigning keywords and information branches to the database entry;

segmenting the multimedia file with into shorter video and audioportions, still frames, and regions within still frames thereby-creatingthe at least a second database entry;

assigning a short title to the at least second database entry;

assigning keywords and information branches to the at least seconddatabase entry;

wherein each database entry's metadata is browsed and/or searched duringthe retrieval process by the user, said metadata ultimately providingthe location and starting position of the multimedia file or portionthereof associated with the database entry.

The method further provides for retrieving the annotated multimediafiles or portion of multimedia files by means of the user querying thedatabase by browsing the database's knowledge tree to find informationbranches that are associated with the desired multimedia files orportions thereof, or by conducting keyword searches.

A preferred embodiment of this invention provides the additional step ofproviding the user with multiple views of the database's contentsincluding at least a knowledge tree view.

In a further preferred embodiment of this method there is provided afurther step of providing a manipulated view of the database'sstructure, the depiction comprising a contorted view of at least twoinformation branches intersecting at nodes matching the search term, andwherein the user can jump up or drop down a level on the intersectedtree to see the classification used in proximity to the intersected nodein one view.

A further preferred embodiment of the invention involves the userselecting individual information branches from the intersected view forfurther analysis.

In a preferred embodiment of the invention, there is an active node ofthe knowledge tree which is associated with a further view depicting thecontents of the knowledge tree at that point, as determined by theinformation branches contained within the database entries whichcorrespond to certain parts of the database's knowledge tree.

In a preferred embodiment of the invention the user browses theknowledge tree by moving up, down and laterally to get to the nextactive node.

In a preferred embodiment of the invention the content view of each nodeon the knowledge tree is updated and kept live with each movement on theknowledge tree by the user.

In a preferred embodiment of the invention, the invention furthercomprises the step of highlighting on the knowledge tree those nodes ofthe tree that contained database entries which matched the search termused to query the database.

In a preferred embodiment of the invention the user can conduct a searchusing a keyword or alphanumeric string over the contents displayed as aresult of selecting a certain node.

The method further provides that when the user locates a multimedia filethrough reference to metadata located in a database entry, the user isable to start playing the segmented portions of multimedia files,particularly audio and video multimedia files, from their startpositions recorded in their database entries and not the true startposition of the multimedia file.

According to a further aspect of the invention there is provided asystem for a large group of users to locate, segment, annotate andretrieve multimedia files wherein access privileges can be modified foreach user or type of user such that only certain multimedia files areavailable to that user or type of user, or that certain functions, suchas adding new users are only granted to certain users or types of users.

According to a further aspect of the invention there is provided amethod for providing targeted advertising for display to users whilstsearches are being conducted, said method comprising the steps of:

identifying information branches and specific nodes on a knowledge tree;

auctioning or selling the right to display advertisements when the userbrowses over a node of the knowledge tree, selects a specificinformation branch, or when the user is viewing a multimedia file thathas associated with its database entry at least one information branchcorresponding to the one purchased by the advertiser;

displaying the advertisements of the highest bidder when the userbrowses over the auctioned node of the knowledge tree, or selects theauctioned off information branch, or when the user is viewing amultimedia file that has associated with its database entry at least oneinformation branch that was purchased during the auction.

According to another aspect of the invention there is a method ofproviding targeted advertising, said method comprising the steps of:

identifying information branches and nodes on a knowledge tree;

creating a database entry for an advertiser which contains at least oneinformation branch which describes the advertising content;

such that when a user is browsing the contents of a particular node orinformation branch the advertisement appears to the user as content.

According to another aspect of the invention there is provided a methodfor providing targeted advertising, said method comprising the followingsteps:

aggregating at least the metadata contained within disparate multimediametadata databases in a central database accessible over the internet;

providing advertisers with the metadata used in each of the disparatemultimedia metadata databases;

the advertisers bidding on the keyword, alphanumeric string,classification term or information branch that they desire, said bidincluding the details of which database users they wish to advertise to;

the database manager accepting the bidder or bidders that maximize theoverall revenue raising capacity of the network;

the database manager communicating with the individual multimediametadata databases that the advertiser has selected for advertising toits users, said communication including directions to associate theadvertiser's advertisement with the keyword, alphanumeric string,classification term or information branch that the advertiser purchasedsuch that users of the database will be presented the advertisement ifthe relevant keyword, alphanumeric string, classification term orinformation branch that the advertiser purchased is found in a searchresult or as a result of browsing.

In a preferred embodiment of this invention the disparate multimediametadata databases provide the centralized database a detailed accountof past searches conducted over the database and the results of thosesearches, these are provided to the advertisers in addition to themetadata so as to assist them in their decision making.

In a preferred embodiment of the invention the centralized databaseaggregates classification terms and information branches used in eachdatabase's knowledge tree which are then passed on to advertisers fortheir review and eventual bidding.

In a further embodiment of the invention there is provided a method forlocating files that are available on the internet, wherein said methodincludes the following steps:

locating the file of interest on the internet;

adding a database entry to the metadata database in respect of the file;

annotating the database entry with metadata including at leastinformation branches which describes the file;

incorporating the metadata into the database including the incorporationof information branches into a knowledge tree;

searching for the file by browsing or searching the metadata databaseknowledge tree;

obtaining the file from the internet once the relevant database entry isuncovered and the URL obtained.

BRIEF DESCRIPTION OF THE DRAWINGS

In order that the invention may be more readily understood and put intopractical effect, reference will now be made to the accompanyingdrawings which each depict preferred embodiments of the presentinvention:

FIG. 1—depicts the devices the invention may be implemented upon

FIG. 2—depicts the basic constituents of one aspect of the invention

FIG. 3—depicts the logical architecture of one aspect of the invention

FIG. 4—depicts one aspect of the invention on one computer (locally)

FIG. 5—depicts one aspect of the invention substantially on a LAN

FIG. 6—depicts one aspect of the invention substantially on a WAN

FIG. 7—depicts one aspect of the invention in which the content ishosted centrally

FIG. 8—depicts a further aspect of the invention in which the content ishosted in a distributed manner

FIG. 9—depicts the different multimedia files managed by one aspect ofinvention

FIG. 10—depicts the file formats supported by one aspect of theinvention

FIG. 11—depicts a knowledge tree according to one aspect of theinvention

FIG. 12—depicts the segmentation and annotation functions of one aspectof the invention

FIG. 13—depicts catalog entry fields

FIG. 14—depicts overlapping segments of a multimedia file according toone aspect of the invention

FIG. 15—depicts the hierarchy of user roles according to one aspect ofthe invention

FIG. 16—depicts annotation and segmentation workflows in one aspect ofthe invention

FIG. 17—depicts the retrieval process according to one aspect of theinvention

FIG. 18—depicts the retrieval process in more detail

FIG. 19—depicts a knowledge tree of classification terms according toone aspect of the invention

FIG. 20—depicts individual information branches of the knowledge tree

FIG. 21—depicts classification terms as slices of a knowledge treeaccording to one aspect of the invention

FIG. 22—depicts a screenshot after conducting a keyword search accordingto one aspect of the invention

FIG. 23—depicts a screenshot after user rises the slider up on theintersected view according to one aspect of the invention

FIG. 24—depicts devices accessing the metadata database according to oneaspect of the invention

FIG. 25—depicts a network of metadata databases according to one aspectof the invention

FIG. 26—depicts a hub and advertiser server implementation according toone aspect of the invention

FIG. 27—depicts various views of knowledge trees according to one aspectof the invention

FIG. 28—depicts a screen shot post keyword search showing advertisementsaccording to one aspect of the invention

FIG. 29—depicts a screen shot after user selects one information branchaccording to one aspect of the invention

FIG. 30—depicts a screen shot after user selects one of the videos inthe search results according to one aspect of the invention;

FIG. 31—depicts a screen shot after the user selects one of theinformation branches according to one aspect of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The growth in the market for internet connected multimedia devices,including computers and laptops, combined with the rise of socialnetworking has its roots in distinctly personal motivations. It is thedesire to publish—whether to express oneself artistically orintellectually—the desire to ‘be heard’ or to initiate online‘conversations’ and receive feedback, to share one's expertise and builda community of ‘subscribers’ to your content, or just to ‘show off’.

What has become known as ‘citizen journalism’ is but one aspect of thisphenomenon. The core motivation here is the same as for ‘publishing’one's views via talkback radio, or in a journal, book or blog. But asmore and more people turn to multimedia as their chosen form ofexpression and the internet as their chosen method of publishing, andwith more and more people consuming this online content, the demand forsimple, intuitive methods for navigating this content is increasingrapidly.

The key to having a flexible navigation system such as this is theexistence of a personal metadata database as its foundation unit. Thisdatabase contains the ‘tags’ that a person has created to allow him toeasily navigate (search, browse and retrieve) content of interest. Asmore and more multimedia content is published, it becomes more and moreimportant to be able to find exactly what one is interested in, from theocean of choices available. This is the value of a contained, personal‘view’. By combining these personal databases to form group views and/orcollective views, an individual can opt to use expanded, richer but alsomore complex views to navigate content, whenever he chooses. Thecreation, use and re-use of personal databases such as these is at theheart of this invention.

A personal metadata database, as described, can represent more value toits owner than its value purely as a superior navigation tool formultimedia.

Since the tags represent the meaning that humans ascribe to particularmultimedia, they are of potential value to advertisers who want to reachpeople who use search terms the advertiser ‘owns’, as they browse,search and view the content that's been described using tags orcombinations of tags in a personal metadata database.

Generally the present invention is concerned with providing systems andmethods for one or more users to locate, annotate and/or retrievemultimedia files. Whilst the present invention will be described withrespect to multimedia files and computers, the person skilled in the artwill recognize the fact that the invention may be implemented in manydifferent devices and situations.

Turning to FIG. 1 the range of potential devices in which the inventionmay be implemented is depicted. Such devices include, laptop computer10, set top box 12, internet connected navigational units 14, gamesconsoles 16, network attached storage devices 18, and storage devicesgenerally, portable music and/or games device 20, video conferencingdevice 22, personal digital assistant 24, and digital mobile telephone26.

Turning to FIG. 2 the essence of the invention is disclosed which isdatabase 30. This database contains only metadata about the multimediafiles of interest. The multimedia files of interest are represented byonline container 32, which is connected to database 30, through theinternet 34. Ultimately the user sitting at computer 36 queries thedatabase 30 via a client side application running on the computer (notshown). The database 30 instructs the computer 26 to obtain themultimedia file of interest from online container 32.

FIG. 3 depicts the arrangement of logical components which in the caseof the present example depicted in FIG. 3 is implemented on twocomputers, a server 40, and a client 42, both containing multimediacontent, in addition to an online storage repository 54 containingfurther multimedia content. On the server side 40, there are three maincomponents, metadata database 44, database manager 46 and a localrepository of multimedia files 48. The database manager 46 controls theaccess to and modifications to the database 44 which contains theinformation about the multimedia files located in local repository 48.However for these interactions to occur, a user needs access thedatabase from the client computer 42 connected to server via a networkconnection 50 including those that run on the TCP/IP network protocol.Specifically, the user needs to operate the database client 52 to querydatabase 44, through the actions of the database manager 46. Thedatabase 44 may contain metadata in respect of multimedia files locatedin repositories 54 and 56 if the contents of these repositories havebeen added to the database 44, via the database manager 46, andrepository 48.

FIGS. 4, 5, and 6 all depict alternative implementations of theinvention. In particular the depiction in FIG. 4 is that of a computer60, which has software that embodies the invention loaded onto itrepresented by the software objects 62, 64, and 66. The depiction inFIG. 4 is a that of a local implementation. All software modules anddata reside on the one computer. The computer 60 does not communicatewith any other computer on any network. Notwithstanding the fact thatthe implementation is local, the logical architecture remains the sameas in the case of FIG. 3. In particular the computer 60 still needs tohave loaded onto it client side applications being the media player 62and database client 66. Further the implementation in FIG. 5 requires adatabase manager 64. In either of the implementations depicted in FIGS.4, 5 and 6, the media files of interest located in the repositories areaccessed by the media player through reference to the media files URLwhich may include both HTTP: and FILE: protocols.

FIG. 5 depicts an alternative arrangement, one which is predominantlyLAN based. In this configuration the user operates computer 72 whichcontains the database client application, to query metadata database 74,which sits on the same local IP network as repository of multimediafiles 76. The figure also depicts internet gateway 78 and a repositoryof multimedia files 80, located on the WAN. This configuration might besimilar to what one would expect a small to medium media company to needto implement to manage their content stored on their corporate LAN withthe occasional need to access materials on the greater WAN or internet.

FIG. 6 depicts a further arrangement of components including computer82. However in the present example, the metadata database 84 is locatedon the other side of internet gateway 86 which places it on the internetor WAN.

FIGS. 7 and 8 depicts two further alternative embodiments of theinvention. The figure depicted in FIG. 8 is representative of theexamples provided in FIGS. 6 and 7 in that the metadata database is keptseparate from the repositories of multimedia files. As can be seen whenregard is had to FIG. 8 the only data being saved to the metadatadatabase is metadata. No copies of the multimedia files are maintainedwithin the database for later retrieval. Further, once the databaseserver instructs the client application resident on computer 82 where tofind the multimedia file in question the client application goes andobtains it directly from the relevant online repository and not theserver 84.

The depiction in 7 however is the opposite. In this scenario, when amultimedia file is requested during step 88 it is returned not only tothe client application during step 90, but also the metadata databaseduring step 92 such that when the file is requested during step 94, thefile can be delivered centrally from the server during step 96 (orretrieved locally if it is still in its cache).

FIG. 9 depicts the types of multimedia files that one embodiment of theinvention can manage. Presently the invention has been designed tooperate with audio, still images, and video (either with or withoutsound). The person skilled in the art however will realize that thepresent invention has application outside of these three distinct areasand that these other areas fall within the scope of this invention. Forexample, the invention could equally be applied to collections of textor indeed, executables or other program files located on the internet.The file formats that are currently accepted by the present inventioninclude those listed in FIG. 10.

FIG. 11 depicts the way in which multimedia file metadata is arranged,being a hierarchical structure, according to one aspect of theinvention. On the diagram, the knowledge tree 100 is depicted. It iscomprised of nodes 110 which are classification terms taken from eitherthe global or personal dictionary of the user. These terms are organizedin a hierarchy such that the hierarchy of terms conveys more semanticmeaning than any individual classification term or collection of termswhere the relationship between the terms is not as apparent as it is ina hierarchy. It is this semantic meaning and attribution of the meaningto individual media files 105 and segments thereof. The individual mediafiles do not form part of the knowledge tree 100. For example node 110of the knowledge tree 100 contains a collection of multimedia filesincluding HeatA.mov 130. Whilst not shown on the diagram, each node ofany level can “contain” multimedia files, not just the lowermost one. Inreality however, the node only contains a reference to the databaseentries that are associated with the multimedia file or segment thereof.

The user of the present invention is able to associate with the databaseentry one or more information branches 120. Information branches aresegments of knowledge tree 100. As indicated in the diagram, informationbranch 120 begins with the root node, “Olympic Sports” and proceeds downuntil “100M Mens”. During annotation, the user selects the lowermostclassification term, in the present case, 100M Mens 110, and drops itonto the database entry. The database manager then associates thatmultimedia file, in the present case, file 130, with the informationbranch 120 such that during browsing of the database when the userchecks the “contents” of node 110, database entry referring to file 130is found.

One of the more powerful features of the invention is the ability forusers to provide annotation on many different levels (of segmentation).FIG. 12 depicts the various levels available to an annotator. Thepresent invention provides for the annotation at the repository level140 wherein all multimedia files contained within a certain repositorywill be branded with the annotations provided at this level. Theinvention further provides for the annotation at the “whole of work”level 142. The user may also take individual segments 144 or even pullindividual frames 146 from video files, or alternatively the user mayhave still images needed annotating which can be achieved at this framelevel 146. Users of the invention may also decide to annotate a regionwithin the frame 148 by means of drawing a polygon around the region.

FIG. 12 also depicts the metadata types applicable at each level ofannotation. At the first level, that of the repository 140, the onlymetadata applicable is that of location, specifically, URL's. The secondlevel is the whole of work level 142. This is the level with thegreatest amount of annotation options available to the annotating user.In particular the annotating user can associate (1) classificationterms/information branches, (2) keywords and (3) catalog entry metadata.At the segment level 144, the annotating user can add start and stoptime information, duration information, classification terms/informationbranches and finally, keyword information. The still frame level 146 andregion within still frames 148 both have the capacity to associateclassification terms/information branches, keyword information andinformation relating to the point in segment/time the frame/regionrelates to.

FIG. 13 depicts the catalog entry fields available to the user in oneembodiment of the invention. These fields are only applicable to thewhole of work level 142 as depicted in FIG. 12.

FIG. 14 depicts the segmentation of movie or video files. In thisfigure, video file 150 is segmented into segment A 152, segment B 154and segment E 156. Because the multimedia files according to thisembodiment of the invention are not required to be duplicated oractually segmented but rather are virtually segmented, the segments 152through 156 can be overlapping in their content. Indeed, the concept ofvirtual segmentation also provides for a further additional benefit,deep linking, and the avoidance of copyright infringement. The presentinvention provides that when a user segments a video file into shortersegments which later are sought to be retrieved, the user is initiallypresented with the shorter segment and not any of the preceding materialthat would have been viewed had the whole multimedia file been played tofind the desired content. The invention achieves this by automaticallypopulating the time in and time out fields of the segment. Whenretrieved the database manager provides the user with a link that takesthe user to that point in time of the original video file withoutrequiring a second smaller copy of the segment of the file to beresident in a repository.

A further feature of a preferred embodiment of the invention is thefeature of multiple levels of access to the database. FIG. 15 depictsthe 4 roles currently provided for by the invention. Further the tableprovided in FIG. 15 depicts the differences in roles carried out by eachtype of user. In addition to these 4 levels of users, there are variouslevels provided for granting differing levels of security or access tothe administrator, librarian, annotator and searcher roles. The securitylevels are such that they can be applied so that users can be restrictedto certain repositories, or restricted in other ways, includingrestrictions on functionality. For example, certain users may beprevented from viewing certain material based on the user's location inthe world, place in an organisation or age. This may be due to theoperation of copyright or censorship laws by certain states orjurisdictions. Further, in a situation where there are multiple metadatadatabases in operation by various content aggregators and owners acrossa network, or the internet, a user may be permitted to log into certaindatabases, but not others.

FIG. 16 depicts the process carried out, according to one embodiment ofthe invention, to virtually segment and annotate multimedia fileslocated on the network. The first step is to determine whether therepository (the volume mounted on the network) is known to the metadatadatabase manager. If the volume is registered then the user provides themultimedia file's path, filename and short title to the metadatadatabase manager. This information is sufficient for a database entry tobe created and associated with the multimedia file in question. Thedatabase entry can be annotated with any of the fields as permitted anddepicted in FIG. 12, including keywords, alphanumeric strings, andinformation branches pruned from the databases' knowledge trees. A usercan takes matters further by virtually segmenting the larger multimediafile into clips of shorter duration or even frames or regions within aframe. In each of these cases the resultant segments, frames, or regionswithin frames are associated with distinct database entries (by themetadata database manager) to that of the database entry that relates tothe whole multimedia file. Once separated from the original databaseentry, the subsequent database entries can also be annotated with any ofthe fields as permitted and depicted in FIG. 12, including keywords,alphanumeric strings, and information branches pruned from thedatabases' knowledge trees.

Thus, the user can use any or all of the information entered by theannotating user to locate the desired multimedia file of interest. Thisallows the user to conduct keyword searches over (1) catalog entryfields which apply only to the whole of work level, (2) classificationterms/information branches, and (3) keywords.

FIG. 17 depicts the broad strategy used by a user to obtain themultimedia file that they desire. According to the present embodiment ofthe invention, there is provided an iterative process whereby the usersearches by utilising keywords or browses the knowledge trees insuccession until the entry corresponding to the multimedia file inquestion is found within the retrieved results. FIG. 18 depicts asimilar process in algorithmic form. In the process so describedtherein, the user first selects a node of the knowledge tree or akeyword to conduct a first level search over the entire database ordatabases, then determines whether the content of interest is containedwithin the node or the results returned by the search, if not, the usereither browses the tree to arrive at a different node, or conducts afurther keyword search to obtain a different set of search results, theprocess being repeated until the multimedia file desired is located.

FIG. 19 depicts a hypothetical knowledge tree 160, of a metadatadatabase. The knowledge tree is comprised of a hierarchical structurewith “Hollywood” at its root. When the knowledge trees are examined forthe occurrence of the term “Simpsons” the information branches 170 areobtained from the knowledge tree 160. As can be seen in FIG. 20 (whichis a depiction of the individual information branches of the knowledgetree) the Simpsons feature in many of the information branches of theknowledge tree, at various levels.

When the information branches below are arranged such that theyintersect at the common classification term “Simpsons” and visualised in3 dimensions the structure depicted in panel 210 of FIG. 22 is arrivedat. FIG. 21 depicts 2 dimensional cross sectional views of this threedimensional form. Slice 200 is the slice made at the intersection, andthus only depicts one information branch existing at that level. Howeverif the user takes a slice one level above the intersection 180 the slice210 is arrived at. Slice 210 depicts all of the information branches asthey branch out from “Simpsons”. Slices 220 to 240 represent theinformation branches at those levels above the intersection 180, andslice 250, the one level below the intersection of the classificationterm “Simpsons”.

FIG. 22 depicts a sample screen shot according to one aspect of theinvention. The screen is divided into a number of different areas orpanels. Panel 200 comprises the knowledge tree view, which is in thepresent case, a highly complex and populated knowledge tree, panel 210,the intersected branch view, panel 220 the keyword search panel, panel230 the search results list view, panel 240 the views selector and panel250, the navigation panel. The intersected tree view is comprised ofthree elements, the 3D structure 212, vertical slider 216 andinformation branches 214 that exist at the level set by the slider 216.The search result list view 230 provides a list of database entries 232.The embodiment of the invention depicted in FIG. 16 is highlycustomisable. The user can select whether to conduct a keyword searchover the current node, or whether to conduct the search over a largerportion of the tree or even the whole tree by selecting the appropriateradio dialog box in the keyword search panel 220. In addition the usercan restrict searches to classification terms/information branches,catalog entries or keyword fields.

In FIG. 23, the user has taken the results provided by a search for“Simpsons” as depicted in FIG. 16A and further queried the metadatadatabase. In the present case the user has taken the intersected treeview and slid the slider 216 up one level which revealed classificationterms 214. The user has then selected “animated” 218. By selecting“animated” 218, the user causes the branch of the knowledge tree to bedisplayed in the knowledge tree view 200. Further, all multimedia filesassociated with the node “Animated” are displayed in the search resultsview.

FIG. 28 depicts various representations of knowledge trees structurespossible in accordance with the teaching of the present invention. Ashierarchical information can get difficult to represent in a traditionaltree view as depicted in FIG. 19, different ways of displaying the datahave been employed such that no matter how complex a data set, there isfirstly, a way of visualising it, and secondly, a way to zoom in andtraverse it.

FIG. 24 depicts the various different devices that are encompassed byparticular embodiments of the invention. In all of these cases there isa metadata database and manager 260 and IP network 270. The most usualdevice for the invention to be implemented upon is what is known as apersonal computer, which are often sold in desktop and notebookconfigurations. The personal computer can be a client in one of a numberof ways. Computer 280 has installed upon it a web enabled executableprogram which acts as database client and is capable of exchanginginformation with the metadata database and metadata database manager260. This program would need to be installed by any user wishing toaccess network multimedia files described in the metadata database. Theprogram can be developed into two or more versions. One version of thesoftware is accessible by all levels of users including librarians andadministrators and facilitates the management of users and themaintenance of the database, this is called the thick client. In oneembodiment of the invention the thin client on the other hand isdesigned only for the searcher user type and thus no annotation ordatabase management functions are included. The only function that thisversion facilitates is retrieval of already annotated multimedia files.

A further embodiment of the invention involves the database beingaccessed through a common web browser such as Firefox, Safari orInternet Explorer as depicted by computer 290 in FIG. 17. As webbrowsers are bundled with computers pre-loaded, any user should be ableto access the metadata database and manager 260 through the common HTMLmediated interface. Both fully functional and search only web interfacescan be developed to access metadata database and manager 260, includingthose that incorporate AJAX technology.

A similar approach can be taken with mobile telephones 300, and personalmedia players (not shown). Most modern mobile telephones and PDA's havethe ability to load executable program files upon them in the form ofJava, Windows Mobile CE, Symbian, and more recently Mac OS Xexecutables. In such scenarios the mobile telephone or PDA can beequated with computer 280 with software loaded onto it. Equally however,most modern mobile telephones, PDA's, and personal media players 310such as the iPod and PSP also feature HTML or web browsers. Thus theymay be able to access the database and manager 260 through a browsertype interface as in the case of computer 290.

Many household appliances that feature the ability to connect to theinternet/IP network and display multimedia files would also becandidates for embedding within their firmware a compatible metadatadatabase client. FIG. 17 depicts an IP enabled portable media playerdevice 310, set top box device 320, and games machine 330 as beingsuitable candidates for inclusion of the database client in theirfirmware, or indeed, as in the case of mobile telephones and somepersonal media players 310, through incorporated web browsers.

Indeed the range of devices is really only limited by their ability toaccess the internet/network and display multimedia files. Other devicesthat may be suitable would include video conferencing equipment, routersand network attached storage devices, in car navigation display units,internet kiosks and in car/plane entertainment systems. Further, thesearch and retrieval function depicted in FIGS. 22 and 23 need not beimplemented in any client device. Indeed, in one embodiment of theinvention, there is provided an open API for interaction with themetadata database such that users may develop their own mash upapplications that utilize the power of the metadata database. Forexample, with respect to the in-car navigation system, it will generallybe aware of its location via GPS. Further, the structure of a metadatadatabase, and importantly, the knowledge tree contained therein might besuch that it has branches that correspond to different GPS locations. Asthe user passes through areas in which there are associated databaseentries (given the GPS co-ordinates) certain videos and/or audio can bedisplayed on its display and through its speaker. Such technology wouldbe ideal for the provision of location specific advertising. A furtherembodiment not involving search and retrieval functions may involve aset top box that has loaded within it firmware that recognizes the typeof program being watched (previously annotated) and which searches forsimilar content over the internet for the viewer whenever a button ispushed on the remote, or which searches for advertisements that havebeen programmed to appear in conjunction with certain programs, eg. BMWadvertisements during “Top Gear” a motoring television program.

FIG. 25 depicts a many metadata database implementation of theinvention. FIG. 25 depicts two standalone computers 350, 370 containingmetadata databases 380, 390, connected through the internet 360, whichis in turn in connected to two other tagmotion databases 400, 410. Inone embodiment of the invention there is provision for the databases toexchange information by means of exporting database entries anddictionaries of classification terms. This can be done manually orautomatically between users in a group where permission exists betweenthem for this to happen. In this way, they create a group taxonomy whereeach benefits from the efforts of the other, enriching the metadataincluding the ‘global (or collective) dictionary terms’, thus making thesearch process more and more intuitive and above all, precise. Indeed,depending on whether the user had the necessary privileges, the user canlog into any one of the databases for search and retrieval. If found,the user might decide to export the entry into their own database, orthe dictionary terms into their own dictionary. Then, the user maychoose to import metadata such as catalog entries, dictionary(classification) terms and storyboards (containing metadata for segmentsof video files, for example) into their own database. These imports canbe from other personal databases or from the collective one. In thisway, one's own personal database or ‘view’ can be enriched as desired,by exploiting the annotations of others.

In one embodiment of the invention it would be possible to provide asystem whereby all users are granted librarian access which means thatanyone could add whole of work database entries and additions to theglobal dictionary from which classification terms appearing on theknowledge trees are selected. In such an embodiment the system wouldprovide for an online “wiki” in which the experience of the group usingthe wiki is affected by each of the users participation. In such ascenario, potentially involving millions of users as in the case ofyoutube.com or myspace.com websites, content would be rapidlycategorized according to potentially millions of different points ofview, thereby creating the rich, metadata-dense, database which in turnresults in the formation of a detailed knowledge tree in which it islikely you would find multimedia files of interest.

FIG. 26 depicts a similar array of metadata databases as depicted inFIG. 19. In particular there are databases 410, 420 and 430 which areall connected the internet and through it to advertising hub 430.Advertising hub 430 is in turn comprised of database 440 and databasemanager 450. In this embodiment of the invention the metadata fromparticipating metadata databases is uploaded into the database 440 wheretheir database entries are analysed and information branches extractedtherefrom. Advertisers 460 are then able to log into the hub 430 throughthe manager 450 and select individual keywords, information branchesand/or nodes on knowledge trees for purchase or bidding in relation towhich their advertisements would be played.

FIGS. 27, 29, 30 and 31 depict the interaction a user has with anadvertiser who has purchased either keywords or information branchesused in the metadata databases' entries. FIG. 27 depicts the samescenario as depicted in FIG. 22, that of a user who has just conducted akeyword search using the word “Simpsons”. In the present case the userhas not yet selected an individual information branch and thus the basisfor serving the advertisements would be the word “Simspons” or theexistence of one or more particular information branches. As semanticmeaning is caught up in an information branch they can be used toprovide more targeted and contextually relevant advertisements.

FIG. 29 depicts the next step in which the user uses slider 216 to go upone level/slice and selects the “animated” information branch. As theuser has selected an individual information branch this can be used asthe basis to serve up further contextually related advertisement 480.

FIG. 29 also highlights another potential advertising possibility inthat positions on the metadata database's knowledge trees can be soldfor advertisers to upload content in respect of their goods andservices. In FIG. 29 advertisement 490 has been inserted atHollywood:TV:Comedy:Animated, the active node. Such positions may beused to feature content including promotional or traditional advertisingvideo, or it may simply act as a link to a website or other position onthe knowledge tree where the advertiser may have more of a presence.Indeed, such an approach is likely to facilitate the development ofmetadata databases in which advertising and promotional content is thesole content of the database. In such case the knowledge tree andassociated search methods could result in a video yellow pages in whichadvertisers place their advertisements at particular places on the treeas opposed to being arranged alphabetically by reference to the goodsand services they supply.

FIG. 30 depicts a screenshot of an embodiment of the invention where theuser has selected one of the Simpsons videos from FIG. 29 and requestedit to play. In this embodiment the search list pane is replaced with amultimedia player and the reproduction of the metadata recorded inrespect of the multimedia file. When the single database entry isselected, the intersected tree view changes to reflect the 3D structureof only those information branches associated with that particulardatabase entry. Further, advertisements 500 are served up with themultimedia file as determined by the remaining information branches andkeywords contained within the various metadata database fields availablefor that type of entry. As there are fewer branches at this point, theadvertisements will be more targeted than those when there were manymore information branches associated with a given view. However,notwithstanding that there are fewer intersecting branches, there arestill more than one, and therefore no one information branch of theknowledge trees is shown in the knowledge tree pane 200, but rather thewhole tree is shown with highlighted “hits” either being informationbranches of the knowledge trees or nodes in which there are databaseentries that have the searched for keyword in their fields. FIG. 31differs from FIG. 23 as the user selected one particular informationbranch 510, which then has the effect of highlighting that informationbranch of the knowledge tree in tree view pane 200 as depicted. Also asthe intersected tree view has been used to select one particularinformation branch, the same branch of the knowledge tree is highlightedin the pane above. Further, as only one branch has been selected, thesemantic meaning of the branch should provide for the serving of evenmore context specific advertisements.

In an alternative embodiment of the invention, rather than the usernavigating through a 3D representation of the knowledge tree with theuser pausing to see what is contained at each node of the tree, the userin this alternative embodiment actually navigates a 3D space in whichthe content rather than metadata descriptors is actually traversed. Insome embodiments you may have intersected information branches beingrepresented by the content associated with the branches. In otherembodiments it may be the structure of the whole tree which isrepresented by content (that is playing or has representative videoframes depicted) by which a user navigates themselves. Different audioor video content will come into view as the user moves through thelandscape. That is, it will have the effect of flying through thecontent in a 3 dimensional space. As this happens, the content beingdisplayed changes according to underlying information branches (that arehidden from view), and what they represent. This effect may also beenhanced by spatial audio controls whereby the sound pertaining to aparticular file becomes louder as the user ‘flies’ closer to it, andsofter as he starts to ‘fly’ away and/or the speed of video playbackbecomes slower as the user ‘flies’ closer to it, and faster as he startsto ‘fly away’. This embodiment of the invention could be provided in theform of an executable program, an AJAX or similar website, or indeed,inside existing three dimensional environments such as Second Lifedeveloped by Linden Labs.

The person skilled in the art will appreciate that whilst this inventionhas been described in respect of audio, images and movie files locatedon the internet, the invention is equally applicable to other items ofinterest located on the internet including documents including HTMLdocuments, information contained in documents, executables, indeedanything of interest that can be located by a URL and filename.

What is claimed is:
 1. A method for distributing multimedia files, orportions thereof, each multimedia file having a true start point and endpoint, the method comprising: identifying a URL of a multimedia file inan online distributed digital repository; creating a first databaseentry for metadata associated with the identified URL; assigningkeywords and classification term hierarchies to the first databaseentry; creating at least a second database entry by virtually segmentingthe multimedia file into a segmented region within an individual frame,the individual frame when visually presented having a whole areadimension extending to the outer confines of the individual frame, thesegmented region having an area dimension less than the area dimensionof the individual frame, the virtual segmentation of the multimedia fileincluding assigning one or more of a keyword and a classification termhierarchy to the at least the second database entry; storing the firstand second database entries in a metadata database remotely from theonline distributed digital repository; receiving one or more queriesincluding one or more of classification term hierarchies and keywords;presenting results of a keyword query three-dimensionally on a screendivided into panels, one of the panels being a panel to navigate theresults of the query; and retrieving the segmented region of theindividual frame from the online distributed repository.
 2. A system fortagging a region within a frame of a digital video file stored in adistributed manner online, the system comprising: a metadata databaseconfigured to store metadata related to the video; and a processorconfigured to generate a metadata database entry in said metadatadatabase that includes identification of a segmented region within aframe of the video, the frame when visually presented having a wholearea dimension extending to the outer confines of the frame, thesegmented region having an area dimension less than the area dimensionof the frame, the identification including a point in time, saidprocessor being configured to provide a link to the point in time in thevideo file, said processor being configured to assign a classificationterm to the database entry, the classification term being organized in ahierarchy compared to other classification terms in said metadatadatabase, said processor being configured to permit a user to conduct akeyword search of the hierarchical classification terms in said metadatadatabase to retrieve the segmented region within the frame of the video,said processor being configured to present results of the keyword searchthree-dimensionally on a screen divided into panels, one of the panelsbeing a panel to navigate the results of the search, and said processorbeing configured to display the segmented region of the video fileassociated with the link.
 3. The system of claim 2, wherein saidmetadata database is configured to store only metadata.
 4. The system ofclaim 2, wherein the region within the video frame is defined by apolygon.
 5. The system of claim 2, wherein said processor is configuredto generate a metadata database entry that includes fields for starttime, end time, duration, location, short title, and keyword.
 6. Thesystem of claim 2, wherein the database entry is associated with aseparate database entry relating to a whole of work entry that includesa URL to the video file.
 7. The system of claim 2, wherein said metadatadatabase is configured for access via an applications program run on aclient computer.
 8. The system of claim 2, wherein said metadatadatabase is configured for access via an applications program run on amobile communications device.
 9. A method for tagging a region within aframe of a video file, comprising: populating metadata fields of ametadata database with a file name and a start time of a video file, themetadata database being remote from an online distributed digitalrepository; segmenting frames of the video file into segmented regionswithin the frames, including assigning a point in time with eachsegmented region, each frame when visually presented having a whole areadimension extending to the outer confines of each frame, each segmentedregion having an area dimension less than the area dimension of theframe containing the segmented region; providing a link to the point intime in the video file containing the segmented region to retrieve thesegmented region; assigning a classification term to each segmentedregion; arranging the classification terms in a hierarchical structure;permitting a keyword search of the classification terms; presentingresults of the keyword search of the classification termsthree-dimensionally on a screen divided into panels, one of the panelsbeing a panel to navigate the results of the search; and retrieving thesegmented region of the video file based on the keyword search.
 10. Themethod of claim 9, wherein the segmenting of frames includes selecting aregion of a frame of the video using a polygon drawing tool.
 11. Themethod of claim 9, wherein the file format of the video file is MPEG-4.12. The method of claim 9, further comprising accessing, with anapplication on a client computer, the metadata database to permit a userto modify an entry on the metadata database.
 13. The method of claim 9,further comprising accessing, with an application on a mobilecommunications device, the metadata database to permit a user to modifyan entry on the metadata database.
 14. The method of claim 9, whereinthe segmenting is virtual.
 15. The method of claim 9, further comprisingpresenting results of a keyword search of the classification terms asvideo content shown three-dimensionally.
 16. The method of claim 9,further comprising displaying the hierarchical structure ofclassification terms in a tree structure.